#12345#####################

testSuite
PlugIns Mar10 Timm

Test suite

``!testSuite``

#12345#####################

plugIns
PlugIns Mar10 Timm

Plugins

The following code was generated using:

  <pre>
  &#96;&#96;!shell; ls -lsa &#96;&#96
  </pre>

This generates:

<pre>
``!shell;ls -lsa``
</pre>

#12345#####################

isNumString
Strings Mar10 Timm

Checking if a string contains a valid number

Problem
=======

How to check if a string contains a valid number?

Solution
========

Regular expressions will do the job. 

 function isNumString(x) {
	return x ~ \
      /^[+-]?([0-9]+[.]?[0-9]*|[.][0-9]+)([eE][+-]?[0-9]+)?$/
 }


   function IsNumStringDemo() {
       print "isNumString " isNumString("23.22E-100")
       print "isNumString " isNumString("a b")
   }

Author
======

Tim Menzies 

#12345 #############################

seed
Numbers Random TimM

Controlling Random Number Generation

Problem
=======

When doing any testing or simulation using any code that generates random numbers,
it is important to be able to reproduce the sequence of "random" choices that lead
to a particular result. This is useful when, say, trying to debug some crash.

Solution
========

Define a global _Seed_ that is to be used to seed the random number generator.
Also, write a _seed(n)_ function that can reset that
seed to a locally set value.

 function seed( seed0) {
      seed0 = seed0 ? seed0 : systime() + PROCINFO["pid"]
      srand(Seed ? Seed : seed0)
 }

Notes
=====

The above code can be called two ways:

+  _seed(1)_ to change the seed to something you can control (in this case, 1);
+  _seed()_ to select some random seed.

Author
======

Tim Menzies

#12345########################################

barph
Strings Timm Mar10

Print a String to Standard Error

Problem
========

While converting input to output, it is often
useful to print notices to a third stream. 
For example, on error cases, it is standard
to print to the UNIX standard error stream.

Solution
========

warning
-------

The _warning_ function prints to standard error. 

 function warning(str,    exitCode) {
     exitCode = exitCode ? exitCode : 1
     print "#"Program" warning: " str   >> "/dev/stderr";
     fflush("/dev/stderr");
	 Patience--
	 if (Patience == 0) 
	 	exit exitCode
 }

_Program_ is an optional string that is prepended to each warning.

_Patience_ is an optional error counter. If set, then it ever gets to zero then we run
out of patience and exit. If not set, then _Patience--_ will send it negative so it will
never be zero so we never run out of patience.

barph
-----

The _barph_ function prints to standard error and
returns a non-zero error status.

 function barph(str,  exitCode) {
     exitCode = exitCode ? exitCode : 1
     warning(str,   exitCode)
     exit exitCode; 
 }

ok
--

The _ok function checks some code, and calls _barph_
if it is non-zero.

 function ok(status,warning,    exitCode) {
	if (!status)
	  barph(warning,exitCode)
 }

The _notOk_ function checks some code, and calls _barph_
if it is non-zero.

notOk
-----

 function notOk(status,warning,    exitCode) {
	if (status)
	  barph(warning,exitCode)
 }

Author
======

Tim Menzies

#12345######################################

a2s
Strings Timm Sandbox Mar10

Turning an Array into a String

Problem
=======

Converting array contents to a string, perhaps with some
separator character in between each item.

Solution
========

Synopsis
--------

  a2s(array   [,sep,start,stop])

Print the array from _start_ to _stop_.  
Insert the _sep_ string between each item (defaults to ",").
The default behavior is to print the array from position one till end of array.
If _stop_ is not provided, this function peeks at the _array[0]_ to get the
size of the array.

Code
----

The processing of the first item in the array is different to the rest since
there is no separator to print  before item one.

 function a2s(a,sep,  start, stop,  tmp, out,i) { 
     sep   = sep ? sep : ","
     start = start ? start : 1
     stop  = stop ? stop : a[0]
     for(i=start;i<=stop;i++) 
         out = out (i > start ? sep : "") a[i]     
     return out
 }  

Note the use of optional parameters: if _sep_, _start_, _stop_ are not used
when _a2s_ is called, they are set to default values in the first few
lines of the function.

   function A2s(array,n) {
       array[0] = split("john,has,a,long,mustache",array,",")
       print a2s(array, "|" )      == "john|has|a|long|mustache"
       print a2s(array, "|", 3)    == "a|long|mustache"
       print a2s(array, "|", 2, 4) == "has|a|long"
   }

Discussion
==========

Using _array[0]_ to store the size of an array addresses a bug in gawk, prior to version 3.1.8
in which _length_ crashed in functions if passed an uninitialized array. 

#12345#########################################################

basename
Files Timm Sandbox Mar10

Accessing the File Name at the End of a Path Name

Problem
=======

Given a pathname, extract the filename at the very end.

Solution
========

 function basename(path,  n,tmp) {
	n = split(path,tmp,/\//)
	return tmp[n]
 }

    function Basename(    s) {
	print basename("/home/timm/tmp/fred") == "fred"
    }   

Discussion
==========

Assumes that the pathname is separated by a slash. This works on UNIX-style file systems
(including Macs, after version 10.0).

Author
======

Tim Menzies

#12345#####################################

deShell
Files Timm Sandbox Mar10

Sanitizing a String

Problem
=======

Some hacker has inserted shell commands into some string that you think is just
a file name. If you try to _getline_ using that string, then (e.g.) all your
files get deleted.

Solution
========

Remove from that string all the special characters that enable shell commands.

 function deShell(str) {
     gsub(/["`\$;\|&><]/,"",str);
     return str
 }

   function DeShell(     s) {
       print deShell("Tim `rm -rf $HOME/*` Menzies") == "Tim rm -rf HOME/* Menzies"
   }

Author
======

Tim Menzies

#12345############################################

factorial
Numbers Timm Sandbox Mar10

Computing Factorials]

Problem
=======

You want to compute the number of orderings in "_n_" numbers.

Solution
========

 function factorial(n,  out,i) {
     if (n < 1) 
         return 1;
     out = 1
     for(i=2;i<=n;i++) 
         out *= i
     return out
 }

Example
=======

    function Factorial(i) {
	for(i=1;i<=20;i++) 
	    print i " " factorial(i) 
    }

Input... 

  for(I=1;I<=20;I++) 
	print I " " factorial(I) 

Output... 

  1  1
  2  2
  3  6
  4  24
  5  120
  6  720
  7  5040
  8  40320
  9  362880
  10 3628800 
  11 39916800
  12 479001600
  13 6227020800
  14 87178291200
  15 1307674368000
  16 20922789888000
  17 355687428096000
  18 6402373705728000
  19 121645100408832000
  20 2432902008176640000

Discussion
==========

For some awks, gives wrong results for large _n_ (for gawk 3.1.7 on a
Mac, the problems starts at _n=21_). This does not happen with
other awks (like xgawk) that use the
MPFR library (multiple precision floating-point).

Author
======

Tim Menzies

#12345##################################################

no
Strings Timm Mar10 Sandbox

Recognize an Uninitialized String

Problem
=======

Sometimes functions are defined with optional arguments. If such functions are called
			 with less than all their arguments, then that function must
recognize the missing arguments and fill them in with defaults. But how to recognize
the missing arguments?

Solution
========

 function no(x) {return (x == "") }

#12345##################################################

s2a
Arrays Timm Mar10 Sandbox

Initialize an Array from a String

Problem
=======

How to quickly generate a set of key/value pairs in an array.

Solution
========

Synopsis
--------

  s2a(string, array [,separator,blankp])

Divides the input _string_ into the output _array_, separated by
_separator_ (default: ',').  Uses the odd numbered elements as the
keys and the even numbered elements as the values.  If _blankp_ is
set, then the keys and values can contain leading and trailing blanks
(otherwise, those blanks are deleted).

Note that the _array_ is completely reset by this function.

Code
----

 function s2a(str,a,  sep,blankp,  tmp,n,i) {
    sep    = sep    ? sep    : ","
    blankp = blankp ? blankp : 0
    n= split(str,tmp,sep)
    for(i=1;i<=n;i+=2) 
        if (blankp)
            a[tmp[i]]= tmp[i+1]
        else
            a[trim(tmp[i])]= trim(tmp[i+1]);
    return n/2
 }

   function S2a(   a,n,i) {
      n= s2a("abc=1,def=2,ghi=3",a,"[=,]")
	  print a["abc"] == 1
	  print a["def"] == 2
	  print a["ghi"] == 3
   }

Author
======

Tim Menzies

#12345####################################

options
Files Sandbox Timm Mar10

Read Command-Line Options

Problem
=======

Suppose a script uses command-line arguments with flags like _-a_ or,
if an argument is to be set _-x ARG_. If so, then something
has to read those flags, then remove them so that Awk can run
the remaining arguments, as per usual.

Solution
========

The _option_ function revises the two magic Awk variables _ARGC_ (the
number of arguments) and _ARGV_ (the values of those arguments). In
the revised version, all the flags are removed and _ARGC_ is set to a
smaller number (the number of arguments without the flags).  

The following example calls shows _ARGC_ being changed directly. Note that
_ARGV_ is changed inside the function call.  In this example, we
defined flags _-a_, _c_, and _h_. The _-a_ variable is given a default
value (via the syntax _a=1_). The _options_ function assumes that any
flag with an existing value expects a command line argument. So, in
this example, this command line is legal:

  -c -a 21 -h

but this command line flags an error (because there is no argument given to _-a_:

  -c -a -h

Example
-------

 function optionsDemo(opt) { # returns 0 if bad options
      s2a("a=1;c=;h=",opt,"[=;]")
      ARGC = options(opt,ARGV,ARGC)
 }

Code
----

 function options(opt,input,n,  key,i,j,k,tmp) {
   for(i=1;i<=n;i++)  { # explore argstill no more flags
      key = input[i];
      if (sub(/^[-]+/,"",key))  { # we have a new flag
	 if (key in opt)         # if legal flag, change its value
	     opt[key] = (no("a" opt[key])) ? 1 : input[++i]  
	  else 
	     barph("-"key" unknown.")
      } else {
	   i--;
	   break;
	}
   }
   for(j=i+1;j<=n;j++)  # copy to tmp, what's left on the input
       tmp[j-i]=input[j]
   split("",input,"")   # reset the input args
   for(k in tmp)        #  move the copy back to the input args
	input[k] = tmp[k] 
   n -= i
   return n
 }
  
Discussion
==========

The trick in the above is to clear the options from the command-line such that
the rest of Awk can run right. The final two  for-loops perform that task.

This code is less sophisticated that Robbins' standard _getopt_ function. But
_options_ uses no globals where as _getopt_ drops side-effects into several
places.

#12345#####################################################

constants
Variables Timm Mar10 Sandbox

Defining Constants

Problem
=======

How to pre-define a set of standard global constants.

Solution
========

Write a function that defines all the constants.

Code
====

 function constants() {
   Inf    = 10^32;
   Ninf   = -1 * Inf;
   Ee     = 848456353 / 312129649; # 2.71...
   Pi     = 428224593349304 / 136308121570117; # 3.14...
   good to 29 digits
   Sp     = " ";
   Q      = "\"";
   White  = "^[ \t\n\r]*$";
   Number="^[+-]?([0-9]+[.]?[0-9]*|[.][0-9]+)([eE][+-]?[0-9]+)?$"
   _      = SUBSEP; 
 }
 

Discussion
==========

Globals are bad and _N-1_ globals is always better than _N_.  The above code drops concepts into
a global space and imposes constraints on every function in the AWK cookbook. So this above list
should be kept to a minimum.

Note that:
+ Resetting the random number seed should be separate to this process since that gets called  any number of ways.  
+ The above constants are _not_ defined unless this code is called. So if you do not like these settings, write another _myConstants_ function.

All the above are self-explanatory,  except perhaps the "_\__" constant.
This is a convenient shorthand when processing arrays with multiple keys. Suppose we are using  some multi-dimensional
and the number of (e.g.):

+ _cars,green_ is stored at _a\[cars,green,0\]_
+ _colors_     is stored at _a\[colors,0\]_

With the "_\__" constant, the following code can find the number of green
cars and colors, no matter how what is the dimensionality of the keys:
			      
  function howMany (key,a) { return a[key \_ 0] ]

Author
======

Tim Menzies

#12345#####################################################

globals
Variables Timm Mar10 Sandbox

Defining Globals

Problem
=======

How to keep track of all the globals that change value.

Solution
========

Write a function that resets all the globals to some initial values.

Code
====

 function globals() {
     Patience=20
	 Program="ACB"
 }
 

Discussion
==========

Globals are bad and _N-1_ globals is always better than _N_.  The above code drops concepts into
a global space and imposes constraints on every function in the AWK cookbook. So this above list
should be kept to a minimum.

In the above:

+ _Patience_ : is an error counter. Everytime we see a warning, _Patience_ is decremented by one. If we run out of _Patience_, the program quits.
+ _Program_ : is a string to print in front of every warning, error message.

Note that:
+ The above globals are _not_ defined unless this code is called. So if you do not like these settings, write another _myGlobals_ function.

Author
======

Tim Menzies

#12345#####################################################

namingGlobals
Variables Timm Mar10 Sandbox

Naming Globals

Problem
=======

As discussed in [?staticVars Using Static Variables], it is possible to hide variables from the AWK global space
by declaring extra arguments in a function header.

If you accidently forget to do
that, then variables that should have been local go "rogue" and
pollute the global space, with bizarre and confusing consequences.

A simple naming convention lets you quickly find rogue locals.

+ Locals start with lowercase
+ Your our globals are MixedCase starting with upper case
+ This  leaves AWK's globals as ALL UPPER case

Now we can find the rogues by studying the output from PGAWK's _dump-variables_ command:

  pgawk --dump-variables=$HOME/tmp/vars.out  -f yourcode.awk
  egrep -v '^[A-Z]+:' $HOME/tmp/vars.out

Hint
----

To fund where the rogue was set, the following trick works well in many cases. 

If the rogue was, say, "_x_" then the following regular expression should do the trick:

  egreg 'x[ \t]*=' yourcode.awk

Author
======

Tim Menzies

#12345##################################################

o
Arrays Timm Mar10 Sandbox

Printing an Array

Problem
=======

Printing an array is a very common task. How to do it succinctly?

Solution
========

Synopsis
--------

  o(array [,prefix,control])

Options:

- If _prefix_ is given,  print it before each item.
- If _control_ is a positive integer, then print from 1 to _control_.
- If _control_ is zero, the print in any order.
- Else, if the size of the _array_ is stored at location _array[0]_ then print from one to size.
- Else, if _control_ is a string, use it as  an  argument to a UNIX sort command (otherwise, just sort numerically by the _array_ values).

Code 
----

Print one array
+++++++++++++++

o
___

 function o(a, str,control,   i,com) {
     str = str ? str : "array"
     if (control ~ /^[0-9]/)  {
         if (control==0)
             o1(a,str)
         else
             for(i=1;i<=control;i++)
                 print oprim(str,i,a)
     } else
         if (0 in a) # sometimes i store array size in a[0]
	     o(a,str,a[0])         
         else {
             com = control ? control : " -n -k 2" 
             com = "sort " com  " #" rand(); # com is unique
             for(i in a)
                 print oprim(str,i,a) | com;
             close(com); }
 }
				      
The _oprim_ function prints one item, after the prefix string.

 function oprim(prefix,i,a,  j) {
     j=i
     gsub(SUBSEP,",",j) 
     return prefix "[ " j " ]\t=\t [ " a[i] " ]"
 }

Simpler
+++++++
 
oo
__
                                                                                  
When "_o_" is too complicated, use "_oo_".

 function oo(a,prefix, i) {
     for(i in a)
         print oprim(prefix,i,a)
 }

Many prints
+++++++++++

Sometimes, you want to print many arrays; e.g.

  os("Names",names,"Addresses",addresses,"Jobs",jobs)

os
__

Printing many items using the "_o_".

 function os(a1,pre1,a2,pre2,a3,pre3,a4,pre4,a5,pre5) {
     if(pre1) o(a1,pre1);
     if(pre2) o(a2,pre2);
     if(pre3) o(a3,pre3);
     if(pre4) o(a4,pre4);
     if(pre5) o(a5,pre5)
 }

oos
___

Print many items using the simpler list print.

  oos("Names",names,"Addresses",addresses,"Jobs",jobs)
 
 function oos(a1,pre1,a2,pre2,a3,pre3,a4,pre4,a5,pre5) {
     if(pre1) oo(a1,pre1);
     if(pre2) oo(a2,pre2);
     if(pre3) oo(a3,pre3);
     if(pre4) oo(a4,pre4);
     if(pre5) oo(a5,pre5)
 }

Author
======

Tim Menzies

#12345###########################################

rank
Statistics Timm Mar10

Ranking a List of Numbers

Problem
=======

For various statistical tests, asets of numbers have to be converted
to their ranks.  For example, If sample "A" generated five values
(5,7,2,0,4) and sample "B" generates six values (4,8,2,3,6,7) then
these sort as follows:

  Samples A  A  B  B  A  B  A  B  A  B  B
  Values  0  2  2  3  4  4  5  6  7  7  8

This generates a set of ranks (and we use the average rank for when
the values are the same:

  Samples A  A   B   B  A   B   A  B  A   B   B
  Values  0  2   2   3  4   4   5  6  7   7   8
  ---------------------------------------------
  Ranks   1  2.5 2.5 4  5.5 5.5 7  8  9.5 9.5 11

Note that, when ranked in this manner, the largest value have the same
rank, regardless if they were was ten to a hundred times larger.  That
is, such rank tests are not susceptible to large outliers.

Solution
========

Synopsis
---------

  rank(arrayOfNumbers, ranks)

_Ranks_ is a set of key, value pairs wheich each key comes from _arrayOfNumbers_
and each value is the rank of the _key_.

Example
-------

 function Rank(  d,r) {
     split("1,1,1,1,1",   d,","); rank(d,r); o(r,"r1");
     split("1,2,3,4,5",   d,","); rank(d,r); o(r,"r5");
     split("1,1,10,10",   d,","); rank(d,r); o(r,"r110"); 
     split("1,1,10,10,20",d,","); rank(d,r); o(r,"r11020");
     split("5,7,2,1,4,4,8,2,3,6,7",d,","); 
	 rank(d,r); o(r,"ro");  
     print r[1] == 1
     print r[7] == 9.5
     print r[3] == 4
     print r[8] == 11
 }

Code
----

 function rank(data0,ranks,     \
               data,starter,n,old,start,skipping,sum,i,j,r) {
     starter="someCraZYsymBOL";
     n     = asort(data0,data)    
     old   = starter
     start = 1;
     delete ranks;
     for(i=1;i<=n;i++) {
	 skipping = (old == starter) || (data[i] == old);
	 if (skipping) {
	     sum += i 
	 } else {
	     r = sum/(i - start)
	     for(j=start;j<i;j++) 
		 ranks[data[j]] = r;
	     start = i;
	     sum   = i;
	 }
	 old=data[i]
     }
     if (skipping) {
	 ranks[data[n]] = sum/(i - start)
     } else {
	 if (! (data[n] in ranks))
	     ranks[data[n]] = n #int(r+1.4999)
     }
 }

Author
======

Tim Menzies

#12345###########################################

mwu
Statistics Timm Feb10

Mann-Whitney  Tests (paired)

Problem
=======

Check if the ranks found in two populations are significantly different.

Solution
========

 function mwu(x,pop1,pop2,up,critical,                 \
	      i,data,ranks,n,n1,sum1,ranks1,n2,sum2,ranks2, \
	      correction,meanU,sdU,z) 
 {
     for(i in pop1) data[++n]=pop1[i]
     for(i in pop2) data[++n]=pop2[i]
     rank(data,ranks)
     for(i in pop1) { n1++; sum1 += ranks1[i] = ranks[pop1[i]] }
     for(i in pop2) { n2++; sum2 += ranks2[i] = ranks[pop2[i]] }
     #
     meanU      = n1*(n1+n2+1)/2;  # symmetric, just use pop1's z
     sdU        = (n1*n2*(n1+n2+1)/12)^0.5
     correction = sum1 > meanU ? -0.5 : 0.5  
     z          = abs((sum1 - meanU + correction )/sdU)
     # 
     if (z >= 0 && z <= critical) 
	 return 0
     if (up) 
	 return median(ranks1,n1) - median(ranks2,n2) 
     else
	 return median(ranks2,n2) - median(ranks1,n1) 
 }

   function Mwu() {
       print "1,  1 " mwu1(1,1)
       print "0.5,1 " mwu1(0.5,1)
       print "2,  1 " mwu1(2,1)
       print "1,  0 " mwu1(1,0)
       print "0.5,0 " mwu1(0.5,0)
       print "2,  0 " mwu1(2,0)
   }
   function mwu1(mult,up,   pop1,pop2,out,i) {
       mult = mult ? mult : 1
       s2a("1 4.6 2 4.7 3 4.9 "	\
	   "4 5.1 5 5.2 6 5.5 "	 \
	   "7 5.8 8 6.1 9 6.5 "	  \
	   "10 6.5 11 7.2",pop1," ")
       s2a("1 5.2 2 5.3 3 5.4 "	 \
	   "4 5.6 5 6.2 6 6.3 "	  \
	   "7 6.8 8 7.7 9 8.0 "	   \
	   "10 8.1", pop2," ")
       for(i in pop1) pop1[i] *= mult
       out = mwu("a",pop1,pop2,up,criticalValue(95))
       if (out == 0)
	   return "tie"
       else if (out > 0)
	   return "win for a"
       else if (out < 0)
	   return "loss for a"
   }

Author
======

Tim Menzies
			       
#12345###########################################

wilcoxon
Statistics Timm Feb10

Wilcoxon Tests (paired)

Problem
=======

Determine if two populations of numbers (of equal size), are
statistically different.  Compute the difference of each pair (a pair
is two items from the same position in both population), rank those
differences, and check the signficance of those ranked differences.

Solution
========

 function wilcoxon(pop1,pop2,up,critical,	\
		   ranks,w0,w,correction,z,sigma,\
                   i, delta, n,diff,absDiff) {
     for(i in pop1) {
	 delta = pop1[i] - pop2[i]
	 if (delta) { 
	     n++
	     diff[i]    = delta
	     absDiff[i] = abs(delta) 
	 }
     }
     rank(absDiff,ranks)
     for(i in absDiff)  {
	 w0 = ranks[absDiff[i]] 
	 w += (diff[i] < 0)  ? -1*w0  :  w0
     }
     sigma = sqrt((n*(n+1)*(2*n+1))/6)
     z = (w - 0.5) / sigma; 
     if (z >= 0 && z <= critical) 
	 return 0
     else  
	 return up ? w : -1*w
 }
 function criticalValue(conf) {
     conf = conf ? conf  : 95
     if (conf==99) return 2.326
     if (conf==95) return 1.960 
     if (conf==90) return 1.645
 }

   function Wilcoxon() {
       print "1,  1 " wilcoxon1(1,1)
       print "0.5,1 " wilcoxon1(0.5,1)
       print "2,  1 " wilcoxon1(2,1)
       print "1,  0 " wilcoxon1(1,0)
       print "0.5,0 " wilcoxon1(0.5,0)
       print "2,  0 "  wilcoxon1(2,0)
   }
   function wilcoxon1(mult,up,       pop1,pop2,out,i) {
       mult= mult ? mult : 1
       s2a("1 78 2 24 3 64 " \
	   "4 45 5 64 6 52 "  \
	   "7 30 8 50 9 64 "   \
	   "10 50 11 78 12 22 "	\
	   "13 84 14 40 15 90 "	\
	   "16 72" ,pop1, " ")
       s2a("1 78 2 24 3 62 " \
	   "4 48 5 68 6 56 "  \
	   "7 25 8 44 9 56 "   \
	   "10 40 11 68 12 36 "	\
	   "13 68 14 20 15 58 "	\
	   "16 32",pop2," ")
       for(i in pop1) pop1[i] *= mult
       out = wilcoxon(pop1,pop2,up,criticalValue(95))
       if (out == 0)
	   return "tie"
       else if (out > 0)
	   return "win for a"
       else if (out < 0)
	   return "loss for a"
   }
		   
Author
======

Tim Menzies

#12345############################

abs
Numbers Timm Mar10

Find Absolute Values

Return the absolute value of a number.

 function abs(x) { return x < 0 ? -1*x : x}

by Tim Menzies		       

#12345#############################

median
Arrays Timm Mar10

Compute Median of a List of Numbers

Fund the middle of the list of numbers. Assumes the numbers are sorted.

 function median(a,n,   low) {
     low = int(n/2);
     return oddp(n) ?  a[low+1] : (a[low] + a[low+1])/2
 }

by Tim Menzies

#12345#############################

oddp
Numbers Timm Mar10

Check if a Number is Odd.

Returns 1 or 0

 function oddp(n) { return n % 2 }

by Tim Menzies

#12345 ########################

agents
Agents Mar10 Timm

Agents

Unix tools like AWK are filters that run over
streams of data. How can we supporting intelligent
agents in such an environment?

#12345 ########################

normal
Statistics Mar10 Timm

How to Find a Normal Distribution

Problem
=======

You a sampling a normal population.
How to find the mean and standard deviation of that population?

Solution
========

Keep the sum of the sample,
the sum of the squares of the sample, and 
the size of the sample.

 function nkeep(x,m) {
     m["n"]++;
     m["s"]  += x
     m["s2"] += x^2
 }
 function nmean(m) {
     return m["s"] / m["n"]
 }
 function nsd(m) {
     return sd(m["s2"],m["s"],m["n"])
 }
 function sd(sumSq,sumX,n) {
     return sqrt((sumSq-((sumX*sumX)/n))/(n-1));
 }

Example
=======

 function Nkeep(   i,m) {
     for(i=1;i<=30;i++)
         nkeep(i,m)
     print nmean(m), nsd(m);
 }

Author
======

by Tim Menzies

#12345 ########################

normalPdf
Statistics Mar10 Timm

How to Find a Probability  from a Normal Distribution]

Problem
=======

Given the mean "_m_" and standard deviation "_s_" of a normal distribution,
you want to determine the probability of an event within that
distribution.

Solution
========

 function gaussianPdf(m,s,x) {
     return 1/(s*sqrt(2*Pi))*Ee^(-1*(x-m)^2/(2*s*s))
 }

   function GaussianPdf() {
       constants();
       print gaussianPdf(100,10,99)
       print gaussianPdf(100,10,120)
   }

Notes
=====

To be precise, the probability of a continuous feature having exactly
the value x is zero, but the probability that it lies within a small
region, say x+-&epsilon;. Since &epsilon; is a constant that weighs
across all possibilities, we can  cancel it out (and  not be compute it).

Author
======

Tim Menzies

#12345 ########################

normalSample
Statistics Mar10 Timm

How to Sample from a Normal Distribution]

Problem
=======

Given the mean "_m_" and standard deviation "_s_" of a normal distribution,
you want to generate "_n"_ numbers from that distribution.

Solution
========

Using the _box\_muller_ transform. That is,
take two samples from the uniform distribution on the interval
(0, 1\] and map them to two normally distributed samples. The polar
form takes two samples from a different interval, \[-1, +1\], and maps
them to two normally distributed samples without the use of sine or
cosine functions.

No, I don't really understand it either. 

 function nsample(m,s,n,a,     i) {
     for(i=1;i<=n;i++)
         a[i]=normal(m,s)
 }
 function normal(m,s) {
     return m+box_muller()*s;
 }
 function box_muller(m,s,    n,x1,x2,w) {
     w=1;
     while (w >= 1) {
         x1= 2.0 * rand() - 1;
         x2= 2.0 * rand() - 1;
         w = x1*x1 + x2*x2};
     w = sqrt((-2.0 * log(w))/w);
     return x1 * w;
 }

Example
=======

The following code should report means and standard
deviations near 100 and 10 (respectively).

 function Nsample(   a,i,m,samples)  {
     seed(1)
     samples=1000
     nsample(100,10,samples,a)
     for(i in a)
         nkeep(a[i],m)
     print nmean(m), nsd(m);
 }

Author
======

Tim Menzies
